Final Exam¶
For the Ozone data from the R package mlbench try the following machine learning prediction algorithm. Read the paper Feature Selection with the Boruta Package and implement the algorithm. Build prediction model for the Ozone variable. Which features are most important?
- 1. Boruta RandomForest Algorithm
NAME: SAJITH GOWTHAMAN NET ID: ek5282
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import matplotlib.pyplot as plt
%matplotlib inline
In [2]:
df = pd.read_csv("Ozone.csv")
In [3]:
df.head()
Out[3]:
In [4]:
df.isna().sum()
Out[4]:
We find a lot of NaN values associated with the data. For a prediction model, it is advised to take the mean of the values of it's respective columns. As this particular data deals with the temperature, we shouldn't expect sudden ups and downs of the values. Mean value would be suitable.¶
In [5]:
df.mean()
Out[5]:
In [6]:
df.fillna(df.mean(), inplace = True)
Let's check the predicting target's data, whether we can do a classification or regression model.
In [7]:
df.info()
df.describe()
Out[7]:
In [8]:
sns.pairplot(df)
Out[8]:
In [9]:
df.V4 = round(df.V4)
df.V4 = pd.to_numeric(df['V4'])
In [10]:
df.V4.unique()
Out[10]:
In [11]:
##Let's Load X and Y
X=df.loc[:, df.columns != 'V4'].values
Y=df.iloc[:,4].values.ravel()
Boruta Model (with Random Forest):¶
In [12]:
from sklearn.ensemble import RandomForestClassifier
from boruta import BorutaPy
rf = RandomForestClassifier(n_jobs=-1, class_weight=None, max_depth=2)
# define Boruta feature selection method
feat_selector = BorutaPy(rf, n_estimators='auto', verbose=2, max_iter = 30)
# find all relevant features
feat_selector.fit(X, Y)
# check ranking of features
feat_selector.ranking_
Out[12]:
In [13]:
feat_selector.support_
Out[13]:
In [14]:
print('==============BORUTA==============')
print (feat_selector.n_features_)
I was first getting the top 4 important features. After some tuning I was able to get the all the important features. (Same when I ran with R-Studio)¶
In [15]:
features = [f for f in df.columns if f not in ['V4']]
len(features)
Out[15]:
In [16]:
important_features = list()
indexes = np.where(feat_selector.support_ == True)
for x in np.nditer(indexes):
important_features.append(features[x])
print('The Most Important Features Are:{}'.format(important_features))
unimportant_features = list()
indexes = np.where(feat_selector.support_ == False)
for x in np.nditer(indexes):
unimportant_features.append(features[x])
print('The Unimportant Features Are:{}'.format(unimportant_features))
Dataset before filtering to the transformed version:¶
In [17]:
imp_df = pd.DataFrame(df[important_features])
imp_df
Out[17]:
The most important features were found to be 'Unnamed: 0', 'V1', 'V5', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12' and 'V13'¶
In [18]:
#call transform() on X to filter it down to selected features
X_filtered = feat_selector.transform(X)
X_filtered.shape
Out[18]:
In [19]:
X_filtered_df = pd.DataFrame(X_filtered)
In [20]:
X_filtered_df
Out[20]:
Let's check how the boruta model has changed the distribution of the 10 variables with HvPlot
In [21]:
import hvplot.pandas
import holoviews as hv
(X_filtered_df.hvplot(kind='scatter', x='0', y='1', by='0') + X_filtered_df.hvplot(kind='scatter', x='0', y='1', by='1') + X_filtered_df.hvplot(kind='scatter', x='0', y='1', by='1'))
Out[21]:
In [24]:
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn import ensemble
X = imp_df
Y= df.V4
In [25]:
df_rf_clf = ensemble.RandomForestClassifier(n_estimators=23)
df_rf_clf.fit(X,Y)
Out[25]:
In [26]:
predictions = df_rf_clf.predict(X)
Predictions For "Ozone" Variable:¶
In [28]:
predictions
Out[28]:
In [ ]: